Entropy Manipulation of Arbitrary Nonlinear Mappings - Neural Networks for Signal Processing [1997] VII. Proceedings of the 1997 IEEE Workshop

We discuss an unsupervised learning method which is driven by an information theoretic based criterion. Information theoretic based learning has been examined by several authors Linsker [2, 31, Bell and Sejnowski [5], Deco and Obradovic [l], and Viola et aZ[6]. The method we discuss differs from previous work in that it is extensible to a feed-forward multi-layer perceptron with an arbitrary number of layers and makes no assumption about the underlying PDF of the input space. We show a simple unsupervised method by which multi-dimensional signals can be nonlinearly transformed onto a maximum entropy feature space resulting in statistically independent features. 1.0 INTRODUCTION Our goal is to develop mappings that yield statistically independent features. We present here a nonlinear adaptive method of feature extraction. It is based on concepts from information theory, namely mutual information and maximum crossentropy. The adaptation is unsupervised in the sense that the mapping is determined without assigning an explicit target output, a priori, to each exemplar. It is driven, instead, by a global property of the output: cross entropy. There are many mappings by which statistically independent outputs can be obtained. At issue is the usefulness of the derived features. Towards this goal we apply Linsker’s Principle of Information Maximization which seeks to transfer maximum information about the input signal to the output features. It is also shown that the resulting adaptation rule fits naturally into the back-propagation method for training multi-layer perceptrons. Previous methods [ 11 have optimized entropy at the output of the mapping by considering the underlying distribution at the input. This represents a complex problem for general nonlinear mappings. The method presented here, by contrast, is more directly related to the technique of Bell and Sejnowski [5] in which we manipulate 0-7803-4256-9/97/$10.00 0 1 997 IEEE 14 entropy through observation at the output of the mapping. Specifically, we exploit a property of entropy coupled with a saturating nonlinearity which results in a method for entropy manipulation that is extensible to feed-forward multi-layer perceptrons (MLP). The technique can be used for an MLP with an arbitrary number of hidden layers. As mutual information is a function of two entropy terms, the method can be applied to the manipulation of mutual information as well. In section 2 we discuss the concepts upon which our feature extraction method is based. We derive the adaptation method which results in statistically independent features in section 3. An example result is presented in section 4, while our conclusions and observations appear in section 5 . 2.0 BACKGROUND The method we describe here combines cross entropy maximization with Parzen window probability density function estimation. These concepts are reviewed. 2.1 Maximum entropy techniques have been applied to a host of problems (e.g. blind separation, parameter estimation, coding theory, etc.). Linsker [2] proposed maximum entropy as a self-organizing principle for neural systems. The basic premise being that any mapping of a signal through a neural network should be accomplished so as to maximize the amount of information preserved. Linsker demonstrates this principle of maximum information preservation for several problems including a deterministic signal corrupted by gaussian noise. Mathematically Linsker's principle is stated Maximum Entropy as a Self-organizing Principle I ( X > Y ) = hy(V>-hy/,y(Ylx) (1) where I ( x , y ) is the mutual information of the RVs X and Y, and hiJ 1) is the continuous entropy measure [4]. Given the RV (random vector), Y E 'Ji , the continuous entropy is defined as